Neural Networks for Signal Processing

i Abstract In this thesis, methods for optimization of neural network architectures are examined in order to achieve better generalization ability from the neural networks at tasks within signal processing. The feed-forward networks described have one hidden layer of units with tanh activation functions and linear output units. The major topics described in the thesis are: Reducing the number of free parameters in the network architecture by pruning of parameters. Pruning is based on estimates (Optimal Brain Damage) of which parameters induce the least increase in the network performance criterion (the costfunction) when they are removed from the network. Finding methods for estimation of the generalization ability of the network from the learning data set. A generalization error estimate (Akaike's Final Prediction Error estimate) is used for choosing the optimal network architecture among di erent pruned network con gurations. Using methods for on-line tuning of the di erent parameters in the network optimization algorithms. The gradient-descent parameter is set using a second order Gauss-Newton method, while the weight-decay parameters are set using a method based on an optimization of the estimate of the generalization ability of the network. The methods proposed for network optimization are all examined by experiments. Some of the experiments are done on data sets from the literature, so it has been possible to compare the obtained results to what has been reported in previous work. The examples are selected from the areas of classi cation, time-series prediction and model building for non-linear systems. Synopsis (in danish) ii Synopsis I denne afhandling er metoder til optimering af arkitekturen af neurale netv rk unders gt med det form al at opn a bedre generalisationsevne for de neurale netv rk anvendt p a signal behandlingsproblemer. De beskrevne neurale netv rk er af "feed-forward" typen med et lag af skjulte enheder med tanh aktiveringsfunktioner og line re output enheder. De v sentligste emner, der er behandlet i afhandlingen, er: Reducering af antallet af parametre i netv rks arkitekturen ved at fjerne parametre fra netv rket. Metoden (Optimal Brain Damage), der er anvendt til at fjerne parametre fra netv rket, baserer sig p a estimater af hvilken parameter, der for ger netv rkets "performance" kriterium (den optimerede funktionen) mindst, n ar parameteren fjernes. Finde metoder til at estimere generalisationsevnen for netv rkene ud fra indl ringsdatas ttet. Et estimat af generalisationsevnen (Akaike's Final Prediction Error estimat) anvendes til at udv lge den optimale netv rks arkitektur imellem forskellige reducerede netv rks arkitekturer. Anvendelse af metoder til l bende at justere de forskellige parametre, der indg ar i algoritmerne anvendt til optimering af netv rkene. "Gradient-descent" parameteren justeres ved at anvende en anden ordens Gauss-Newton metode, mens "weightdecay" parameteren justeres ved at anvende en metode, der er baseret p a en minimering af estimatet af netv rkets generalisationsevne. De foresl aede metoder til optimering af netv rkene unders ges alle p a eksempler. Nogle af eksperimenterne er udf rt p a datas t, hvor resultater er beskrevet i litteraturen. Dette har muliggjort en sammenligning af de opn aede resultater med, hvad der tidligere er rapporteret. Eksemplerne er udvalgt imellem emner som klassi kation, tidsserie-estimering og estimering af modeller for ikke line re systemer. Preface iii Preface The work presented in this thesis was done as a part of a Ph.D study at the Technical University of Denmark, started in September 1991 and nished in December 1994. The work took place at Electronics Institute, with Lars Kai Hansen, Peter Koefoed M ller and John Aasted S rensen as supervisors. Electronics Institute is a part of the Computational Neural Network Center, CONNECT. I wish to thank my advisors for their support throughout the study. In particular I wish to thank Lars Kai Hansen for his always kind and helpful support and for providing comments and suggestions which improved the quality of the thesis. I also wish to thank all the co-authors of the articles appended (Lars Kai Hansen, Jan Larsen, Anders Krogh, Carl Edward Rasmussen, Peter Salamon, Ole Winther, Jan Gorodkin, Torben Fog, Lars Hupfeldt Nielsen, S ren Holm, Ian Law, Olaf Paulson, Christian Linneberg, and Jan Meyrowitsch), Niels M rch, Electronics Institute and Benny Lautrup, Niels Bohr Institute, CONNECT for their participation in many pro table discussions. Finally, I will like to thank Gina Vianelli who have had a hard job trying to improve the English language used throughout this thesis. The work was granted by the Danish Natural Science and Technical Research Councils through the Computational Neural Network Center, CONNECT. I would like to thank The Otto M nsted Foundation for nancial support in connection with conference participations. Lyngby, December 30, 1994. Claus Svarer